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ABSTRACT 

The purpose of this research was to investigate the 
construct validity of the Enrolled Student Survey (ESS) as applied to 
East Tennessee State University students. It has been suggested that 
cultural differences among students may cause the latent constructs 
underlying student satisfaction to be developed differently. The ESS 
is a state-recommended survey for measuring s tudent sat i s f ac t i on with 
the university. Approximately 2,600 undergraduates were surveyed in 
the spring semester 1993. Constructs proposed by the Tennessee Higher 
Education Commission, a six-factor model, were compared with 
data-based constructs in three data-based models produced by 
exploratory factor analysis and confirmatory factor analysis. Data 
were analyzed using hierarchical confirmatory factor analysis through 
the LISREL computer program. A comparative fit index (CFI) was used 
to set a criterion to accept a valid model. Based on this criterion,, 
the worst fit to the university's data was the six-factor model, and 
the best fit to the data, through the CFI and chi-square analysis was 
the refined model developed through confirmatory factor analysis 
incremental fit comparisons. Reasons the data may not fit the 
six-factor model are discussed. (Contains 5 tables, 1 figure, and 13 
references*) (SLD) 
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Abstract 

It has been suggested that tlie cultural differences between Appalacliian students attending 
East Tennessee State University (ETSU) and students of other Tennessee higher education 
institutioas may cause tlie latent constructs underlying student satisfaction to be developed 
differently. Since performance funding for state institutions is partially based on the aSvSessment of 
student satisfaction, it is important tliat the measurement of s:-isfaction take into account tliese 
cultural differences and constmcts, or both. 

Tlie purpose of this research was to investigate the construct validity of the Enrolled 

Student Survey (ESS) as applied to East Tennessee State University students. The ESS is a state 

recommended standardized survey for measuring student satisfaction with tlie educational 

experience. Approximately 2600 undergraduate students at ETSU were surveyed during the 

spring semester 1993. 

llie constructs proposed by the Tennessee Higher Education Commission (THEC) were 
compared with data-based constructs produced by exploratory factor analysis using SPSS-X. Data 
were analyzed using Hierarchical Confirmatory Factor Analysis in LISREL 7.2. Tliree data-based 
models were compared to the Six-Factor Theory Based Model proposed by THEC. The three 
data-based models were: Six Factor hiductive Model, developed via an exploratory factor aniUysis 
where the number of factors were constrained to six; Ten Factor hiductive Model, generated by an 
unconstrained exploratory factor analysis; and Refined Model, developed u.>ing confiniiatory factor 
analysis incremental fit comparisons. Two criteria were used for assessing the validity of the four 

models: comparative fit index (CFI). Because tests with large sajiiple sizes ixxe likely to 

reject reasonable models, the CFI was used as an alternate to x^- 1 be criterion used for this study 
to accept a valid model was CFI = .90. Based on the CFI criterion, liie model with the worst fit to 
the ETSU data was the Six-Factor Theory Based Model (x^ = 91 8 1 , d.f. = 1 310; CFI = .75). llie 

Refined Model had tlie best fit to the data (x^ = 2604, d.f. = 862; CFI = .94). 

ERIC 
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Since this study is based on only one sample, it was suggested tliat tlie lack of fit for the 
SLx-Factor Theory Based Model may reflect: 1 ) an anomaly in the sample, 2) a difference between 
ETSU students and smdents attending other Tennessee institutions of higher education, or 3) a 
measuring instrument that is of dubious validity. Since the ESS is directly linked to incentive 
funding, further research is warranted on the validity of the instmment to ETSU and other 
Tennessee institutions. 
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Hierarcliical Confirmatory Factor Analysis 
of a Student Satisfaction Survey 

The goals of educational accountability and assessment have influenced tlie planning and 
evaluation of educational programs and curriculum in higher education since the late 197()s, This 
state by state campaign for quality in education began witli a demand by external stakeholders for 
tlie justification of increasing higher education costs (Pike, 1993). The interest in accountability 
during the 1980s gained momentum because of the shift in public policy from allowing 
autonomous management of the higher education enterprise to an increased demand for an 
education partnership between the academy and state government. In 1985, only five states had 
accountability and assessment mandates for higher education. However just five years later, forty 
slate legislatures had passed some form of accountability and assessment mandates (Young, 1993), 

Tennessee is tlie front runner in tlie national assessment movement. Since 1979, 
invStitutions of higher learning in Tennessee have participated in an educational assessment program 
(Banta, 1988). At the crux of this assessment program is a ten component perfonnance funding 
fonnula used by state higher education institutions to evaluate the quality of their education process 
and earn aSvSessment points that equate into additional state funding for the institution. One of tlie 
ten components in the perfomiance formula Ls a student satisfaction component. Within this 
component, currently enrolled undergraduate students of participating institutions are surveyed to 
detennine their opinions and attitudes about the institution (Banta, 1988). Tlie results of tlie 
Enrolled Student Survey (ESS) are analyzed by the Tennessee Higher Education Commission 
(THEC) to determine student satisfaction. 

Tlie ESS is administered during tlie spring semester of every odd year with 1 993 as the 
baseline year. The purpose of the ESS instmnient is to measure a Six-Factor Tlieory Based Model 
of educational experience: student satisfaction, student involvement, personal development, 
learning, major instmction and advising, and major curriculum. The results of each survey are 
summary scored witliin tlie six factors. With eacli survey, tliose institutions wlio improve ilieir 
summary score over the previous survey administration or who score liigher than the state mean 
will receive incentive funding from the state. Based on this connection between the six factors of 
tlie ESS and incentive funding, it is important that tlie validity of the ESS instmnient is tested for 
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each of the participating institutioiLs. 
Statement of the Problem 

llie twenty -tliree institutions of higher learning tliat participate in tlie Tennessee 
perfomiance funding program are a diverse group (Banta, 1988). Each of these participants are 
unique in their educational mission and goals, and in their approach to educating their student 
population: a student population that is as unique to each institution as is the educational mission 
of that institution. East Tennessee State University (ETSU), one of the participating institutions, is 
located in the upper northeast comer of Tennessee. As a regional university, 85% of the student 
population comes to ETSU from counties in east Tennessee. Furthermore, 66% of the ETSU 
student population are residents of the five counties tliat immediately surround the university. All 
of tliese east Tennessee counties tliat send students to ETSU are located in the heart of tJie 
Appalachia Southern Region (Hawk, 1993). 

Tlie impact of tlie Appalachian Region to tlie educational experience of tlie ETSU student is 
currently under investigation by the Center for Appalacliian Studies and Services at ETSU. At the 
very most, growing up in Appalachia and being labeled as a "liillbilly" or "grit" may have such a 
profound impact on tlie self-esteem and confidence of the Appalachian student as to negatively 
effect their success in higher education (Speer, 1993). At the very least, tlie Appalachian region 
with its cultural diversity, close community and family relationships, and economic disadvantage 
may provide the Appalacliian student with such a unique K~ 12 education experience tliat tlie 
subsequent higher education experience is equally unique. According to Speer (1993), Director for 
tlie Center for Appalachian Studies and Services, "some campuses are comin;^ to reiUize that 
Appalachian students may need to be recognized as a minority group because they have been 
denied fair access to educational opportunities and because some of them require special assistance 
to succeed" (p. 21 ). 

With tlie majority of students being sent to ETSU from this unique environnienl, it is 
reasonable to question the validity of the standardized ESS to the educational experience of the 
ETSU student. Because performatice funding for ETSU and otlier state institutions is partitUly 
based on the assessment of student satisfaction witli the ovenill education experience, it is 
iinportmit that the measurement of satisfaction take into account the cultural and educational 
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differences of the Appalachian student. 

Tlie purpose of tWs research was to investigate tlie construct validity of the ESS as it 
applied to the ETSU student population. 
Research Questions 

To explore the validity of the ESS to the ETSU student population, the following research 
questions were investigated: 

1 ) Does the Six-Factor Tlieory Based Model of tlie ESS as utilized by THEC adequaiely measure 
tlie educational experience of the student population at ETSU? 

2) Would anotlier model more adequately depict the educational experience of the ETSU student 
population? 

Literature Review 
Educational Accountability and Assessment 

Prior to the late 1970s, liigher education in America enjoyed a level of autonomy that 
allowed administrators to plan and evaluate education prograirxs and cun iculum without , 
interventicMi from various external stakeholders. Until that time, the only participants in planning 
and assessing education outcomes were those internal stakeholders that had a significant influence 
and control on the direction of tlie higher education institution. Stakeholders external to the 
institution, i.e. taxpayers, government officials, and business leaders, were only indirectly 
involved in tlie liigher education process: taxpayers paid taxes to support the operations of the 
institution; government officials appropriated the tax funding; and business leaders employed the 
students that had successfully completed the education process. Each of these constituent groups 
supplied higher education with inputs and employed the outputs, or both, witliout interference in 
the process (Peters, 1994). 

However, in the early 1980s thLs level of granted autonomy began to diminish, hicreitsing 
education costs lliat equated to higher tuition rates for consumers iuid increased funding by 
government, disillusionment within tlie business community with the higher education product 
(Pike, 1993), and the general malaise of distrust among external stakeholders with the ability of 
higher education administrators to miuiage the education process to a quality outcome (Peters, 
1994), are just a few of the reasons for the concerted demand by external stakeholders for 



CFA of a Student Satisfaction Survey 7 

educational accountability and, subsequently, the demand for assessment of the education 
outcome. 

According to Boyer (1988), there were four primary reasons for the natiomd interest in 
assessment. First, leaders of the assessment movement in the 1980s believed thai educational 
assessment would be a powerful impetus to institutional change, which in turn, would equate into 
increased concern for education quality. It was also argued that educational assessment would 
provide faculty an opportunity and an avenue for reassessing curriculum on an ongoing basis. 
Third, educational assessment was seen as a communication mechanism that would stimulate 
discussion about quality in the education process and outcome. This discussion would involve all 
levels of administration and faculty, and would focus their attention on the needs of the customer. 
Finally, educational assessment would restore the confidence of extemal stakeholders in the 
process of higher education. 

The reasons for pursuing educational assessment in the early 1980s, quickly became the 
goals of educational assessment in the eaily 1990s. Leaders of the assessment movement targeted 
goals of promoting institutional change with assessment infomiation; stimulating interest in faculty 
for sweeping reforms in curricula design and implementation (Ewell, 1994); enhancing 
communication among internal and external stakeholders about tlie quality of the education process; 
placing a focus on customer satisfaction (James, 1994); and restoring public confidence in higher 
education 

However, by the mid-1990s tlie goals of educational assessment have not been realized in 
toto (Ewell, 1994). There are many proposed reasons for this lack of goal attainment; assessment 
infonnation is not disseminated to important decision makers (Ewell, 1994): the fear by educational 
leaders that assessment will lead to homogenization of higher education (Davies, 1991); and tlie 
resentment fell by educators on the intervention of extemal publics into tlieir education domain 
(Peters, 1994). However, the most important reason for nonattainment of goids is the difficulty 
willi designing and developing reliable and valid stiindardized instmments: an instmment design 
tliat can accurately measure hypothetical constructs consistently and accurately on ever>' cmipus 
across the slate regiu^dless of geographic <md demographic differences (Banta, Rudolph, & Vmi 
Dyke, 1994; Young, 1994). A proies .ion that Ls founded on the reliability and validity of 
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measurement Is being asked to make decisions and curricular chajiges based on instruments that 
may be providing suspect data. 
Perfomiance Funding in Tennessee 

llie increase in interest by external stakeholders in educational assessment was a national 
concern that quickly disseminated into a state government issue. Each state government begaji, in 
earnest, to investigate metliods of accountability and assessment for institutions of higher learning 
within their state. In response to the demand for educational accountability, government officials 
in Tennessee developed an educational assessment model for institutions of lugher leaniuig in tlie 
state. At the core of this assessment model was a performance funding fonnula that measured 
specific higher education activities ajid programs for quality in process and outcome, and provided 
a monetary bonus incentive as a reward for quality (Banta, 1988). 

Prior to 1975, assessment of education quality in Tennessee was enrollment-based with an 
emphasis placed on quantity of students as a reflection of the quality of the prograjii. In 1975, the 
decision was made by state policy makers to convert this system of enrollment -based assessment 
into perfomiance-based cissessment. The following year pilot projects were administered on a 
variety of university and college campuses across the stale to test the reliability and validity of 
perfomiance-based evaluation. In 1979, with the successful outcome of the pilot testing, THEC 
presented a guideline of perfonnance criteria to state higher education institutions. By 1983, ;uid 
the tightening of the perfonnance guidelines, THEC committed the state to tlie first five-year 
performance funding cycle (Banta, 1988). 

By 1994, and tlie completion of tlie tliird five-year cycle, perfomiance funding has become 
an integral component in tlie planning, implementation and evaluation of higher education 
programs in Tennessee. Currently, there are twenty-three institutions of higher leimiing in the state 
that participate in tlie perfonnance funding model: four technical institutes, ten community 
colleges, and six universities, to include East Tennessee State University, iind tliree University of 
Tennessee campuses, llie peribniiance funding fonnula consists often components: peer review 
of undergraduate progriuiis; placement review; accreditation; improvement actions by institution; 
student satisfaction surveys; major field tests; mission sj^ecific goals of the institution; general 
education tests; retention mid graduation gO£ils; and minority/other enrollment goals. Perfonniuice 
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on each of tlie ten components is worth up to ten points for a total perfonuance maximum of 100 
points (Banta, Rudolph, & Van Dyke, 1994). 

Each point in the fomiula is tied to a dollar value of bonus incentive for the higher 
education institution. Tlie maximum amount of bonus incentive from the fommla is 5.5% of the 
institution s education and general budget (Banta, Rudolph & Van Dyke, 1994). Every year, each 
institution uses the requirements hi the performajice funding fonnula to evaluate tlie quality of the 
educational outcome. The information gathered through this ongoing assessment is forwarded to 
THEC, whereby, the results of the assessment are analyzed in each of the ten components, points 
are assigned in each component, and the bonus incentive is detemiined. 
Education in Appal acliia 

According to DeYoung (1988), tlie educational preparation of students in mral schools 
differs from the educational preparation of students in urban schools. For example, a stronger 
bond exists between the community and ^he rural school than between the community and tlie 
urban school. Because of this strong bond between community and school, and because of tlie 
traditionally conservative values of families in mral areas, the curriculum of the mral school is 
continually scrutinized by the religious leaders of the coinmunity. Also, teachers in mial schools 
have different occupational interests and career ambitions than those who teach in the urban school. 
Finally, for many mral students college is not an option. 

Appal achia shares these characteristics with other mral schools in America with one very 
important distinction: the severe economic disadvantage of the Appalachian region. Because of the 
poverty gap in Appalachia and the continued decline of the mral economy, there exists m\ 
inequcUity of educational opportunity for Appalachian students that does not compiue with students 
of other rural areas nor students in tlie urban environment (DeYoung, 1988). ^Fhose Appalachiiin 
students who do choose college come to campus academically unprepared for the rigors of higher 
education; socially unprepared for the academic culture, processes, and procedures; and 
emotionally unprepared for tlie ridicule by college associates of tlie Appalachian heritage (St:)ecr, 
1993). 

Interestingly enough, in spite of all the differences between the Appalachimi mial school 
and tlie urbaii scliool, past educational refoniis made in mral education have Ix^en based on 

er|c i u 
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educational research perfoniied in tlie urban school environment. It has only been within tlie last 
decade that educational researchers have addressed the need to research the Appalachian rural 
school and design educational reforms that are specific to tliat environment (DeYoung, 1988). 
Many of the urban school reforms that were incorporated into the rural environment simply have 
not worked Urban refonns, such as the consolidated school and centralized staffing, were not 
confluent with the community /school relationship of Appalacliia. 

How does tlie discussion of the unique Appalachian education experience effect higher 
education? If it is reasonable to assume that there are differences in the K- 12 setting because of 
environmental differences between Appalachia and urban than it is equally reasonable to assume 
that there are differences in the higher education experience of a student population that is 
predominandy Appalachian compared with an urban student population. Educational researchers 
have discovered that urban refonns do not work in the Appalachian rural school, llierefore, it can 
be suggested that urban refomis will not work for a higher education institution tliat is 
predominantly Appalachian. Furthermore, it can be argued that the standardized assessment 
instruments designed by THEC witliin the confines of tlie urban experience may not be valid to tlie 
educational experience of the Appalachian student. 

Metliod 

The ESS instrument, as administered in spring 1993, was a seventy-seven question 
measurement of customer opinion and attitude. The instmment contained eight demographic 
questions and sixty-nine questions directly related to the satisfaction of the student with their 
academic, cultural, aJid social educational experience. Included in those 69 questions, were 16 
questions direcdy related to educational services, such as admissions, financial aid, registration, 
the campus bookstore, and career placement. Because THEC did not use these 16 questions in 
fomiing their six factor model, these questions were excluded from tlie confimiatory analysis in 
this study. Therefore a total of 53 questions were used to detennine the structure of the ESS for 
the ETSU student. Tliese questions were scaled using a three to five point Likert-type scale. 

This instrument was designed by THEC to measure six factors of tlie student's educational 
experience; student satisfaction, involvement, jxjrsonal development, learning, major iiistmction 
ixnd advising, and major curriculum. Each of the 53 questions were targeted by THE( to measure 
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one of the six cons^^ructs. Tliere was one qualitative question at the end of the survey which asked 
students to briefly comment on their experiences at ETSU. (Those qualitative responses were 
analyzed by the Office of Outcomes Assessment at ETSU and are not included in this research.) It 
is important to keep in mind, tliat the ESS did not test a student's level of achievement, but instead 
was, and is currently, a measurement of the student's attitude about their level of achievement. 

EiLSt Tennessee State University is a comprehensive regional university offering more Uian 
100 fields of study to approximately 12,000 students. The typical stud'^^nt attending ETSU in the 
fail of 1992 entered the university with an average ACT score of 20.6, compared with the 
University of Tennessee, Chattanooga ACT score of 21 .8: an institution of comparable size and 
educational mission. Of the nine public institutions of higlier learning in Tennessee, ETSU ranked 
eightli in the entering ACT score of freshmen, with the University of Tennessee, Knoxville ranked 
first witli an ACT score of 23.0 and Tennessee State ranked nintli with a score of 18.8. 
Approximately 47% of first-tiine freshman at ETSU were enrolled in developmental and remedial 
CO rses in the fall of 1992, compared with the state average of 33.9%. (Roaden, 1994). 

The target population for the ESS administered at ETSU were all undergraduate students 
enrolled at ETSU during the spring semester 1993. The cluster sampling method was chosen to 
ensure that the sample was representative of the total ETSU population. According to tlie Office of 
Outcomes Assessment at ETSU, 

A random sample was drawn of 25 percent of the 1(K)0 and 2000 level classes in the 
General Education Core and 25 percent of the 3000 and 4(K)0 level classes from each 
college. Tlie sample was drawn from a stratified population of on-campus day classes, all 
evening classes and classes conducted at off-campus sites. ETSU was required to survey 
at least 15 percent of all undergraduate students. To compeasate for attrition and students 
enrolled in more than one surveyed class, 25 percent of all classes were surveyed. Courses 
wliich could not be surveyed in a class setting, such as student teaching/intemship/co-op 
courses were excluded. Tlie surveys were administered by each department in tlie siunple. 
A total of 28.6 percent (2634) of the undergraduate students completed the sur/cy 
(Burnley, 1993, pg. 6). 

As mentioned, ETSU was required by THEC to administer the survey to 15^/{ of M undergraduate 
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students. However, in awarding points in the customer satisfaction component, only ilie data from 
those students v/iti\ 25 or more credit hours at tlie time of the survey, not to include liours taken in 
the spring semester of 1993, were used to determine the satisfaction of students with ETSU. 

In March 1993, the ESS was administered to students that were in tliose classes chosen in 
the sampling procedure. In most cases, tlie class instmctor administered the measurement and 
delivered the completed instrument to the Office of Outcomes Assessment at ETSU. Tliere were a 
few situations in which a representative from the Office of Outcomes Assessment administered! tlie 
exam at the request of the instmctor. Tlie questionnaire administrator gave minimal directions to 
respondents prior to taking the exam to avoid influencing the attitude of the student. Data 
collection for the ESS took approximately one week to complete. After data collection was 
completed, a graduate assistant in tlie Office of Outcomes Assessment manually mput the data into 
a micro-computer database and electronically sent the data to THEC for analysis. A copy of the 
data was kept with the Office of Institutional Research at ETSU for further analysis. 
Analysis 

ITie analysis of the ESS is based on a series of four models. Hie first three models are 

first-order confirmatory factor analysis models^. The first model was derived from the tlieory 

upon which the ESS is based. It is this model (Six-Factor Theory Based Model) that serves as the 
criteria for assessing performance and distribution of incentive fundmg relative to tliat assessment 
(see Table 1 for a description). The second model (Six-Factor Inductwe Model) was developed 
via an exploratory factor analysis where tlie number of factors was constrained to six (see Table 2). 
The third model (Ten-Factor Inductive Model) was generated from an unconstrained exploratory 
factor analysis jthat yielded (based on eigenvalue > 1 ) 10 factors (See Table 3). Tlie fourth model, 
(Refined Model) was developed using CFA incremental fit coniparisoas. The outcome of this 
process was a model with 44 observable indicators (as opposed to 53 for the other models), 16 

first order factors (l^s) and 7 second order factors (^s). Figure 1 provides an in depth summary of 
tiiis model's specification. Additionally, more restrictive "null" models were calculated for both 
Zs (one of rank 53 and one of rank 44)2. 

Two criteria were used for assevSsing the validity of the four models. First, ^2 ^^^^^^ fQj. 

.1 J 
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each model and the difference between the models was used. Because tests with large sample 

sizes and complex models int likely to reject reasonable models, the comparative fit index (CFl) 

was used as an altemative to x^- Tl^is index compares hypotliesized models with a heavily 

restricted (and thus bad fitting) null model, llie rule of thumb is tliat acceptable models must have 
a CFI value of at least .90. 

Results 

Table 4 contains a summary of results. Tlie Six-Factor Theory Based Model, as proposed 
by THEC, was a poor fit to the data, (x^ = 9181, d.f. = 1310; CFI=.75). The Six-Factor 

Inductive Model was a better fit (x^ = 8718, d.f. = 1310; CFI=.77), but this model still 
repiesented a very poor fit. The Ten-Factor Inductive Model represented a substantial 
* improvement over the previous models (x^ = 5739, d.f. = 1280; CFI=.86). Since this third model 

is less constrained than the previous two, it will always fit better in tenns of x^. Yet a comparison 

of this model to the Six-Factor Tlieory Based Model indicates a significant incremental difference 

in temis of x^ (difference in x^ = 3442, d.f = 30, pvalue < .001 ) and in temis of CFl values (.86 

versus .75). Nevertheless, the Ten-Factor hiductive model does not meet the tlireshold CFI value 
for valid models. 

Of the four models, the Refined Model had tlie best fit witli the ETSU data (x^ = 2604, 

d.f = 862 CFI=.94). ITiis fourth model is the least restrictive, and thus it will always fit better 
tlian tlie first tliree models. Yet tlie Refined Model represents a statistically significiuit 

improvement in fit over tlie first tliree models in temis of x^ and CFI value (.94 versus .86) iuid it 

represents a reasonable fit to the data based on the CFI>.90 heuristic. 

Table 5 illustrates tlie relationship between tlie individual questionnaire items and the 
second-order factors. These relationships are indirect effects as mediated by the first order factors. 
Two of the Xs, ESS questions 6 and 58, have double loadings. The remainder of the observable 
indicators load on only a single second-order factor, even though, tlie second-order factors lu-e free 

ER?C I q 
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to correlate among themselves. 

llius, the Refined Model is tlie only model which demonstrates acceptable v^ilidity. The 
poorest fitting model is the Six-Factor Theory Based Model, aiid this model is one upon which 
decision making criteria are based. In other words, funding distributions and other resultant 
outcomes are based upon the least valid model. 

Discussion 

Tlie Six -Factor Tlieory Based Model proposed by THEC does not adequately fit tlie sample 
selected from ETSU. Tlie hierarchical model (Refined Model) developed from this sample 
provides an adequate fit. Since this study is based on only one sample, this lack of fit may reflect: 
1) an anomaly in the sample, 2) a difference between ETSU students and students attending otlier 
Tennessee institutions of higher educr ion, or 3) a measuring instrument that is of dubious validity. 

Several minor differences in the factorial structure of this survey instrument were detected 
between the Six-Factor Theory Based Model proposed by THEC (Table 1) and the Refined Model 
(Figure 1 ) developed in this study. The major difference, however, Ls reflected in the factor named 
'Research* (4 variables) in the Refined Model and the factor 'Involvement' (9 variables) in the Six- 
Factor Tlieory Based Model. Both factors contain four identical variables, llie Refined Model, 
however, does not contain the remaining 5 variables in the 'Involvement' factor. When these, 
questions were entered as an additional first-order factor or the individual questions were added to 

an established factor, the fit as measured by and the goodness of fit function were significantly 

worse. Consequendy, if the mean of the nine variables contained in the Six-Factor Tlieory Based 
Model 'Involvement' factor are used for evaluation of an ETSU student's involvement, more than 
50% of that mean is detemiined by variables that are not related to the construct. In effect, the 
mean \s sufficiently influenced by variables that measure no established factor. Tlierefore, the 
resultant mean of tlie 'Involvement' factor is of questionable validity. Tlie purpose of tliis survey 
was to estal^lisli measures on genenil factors and to evaluate an institution based on improvement 
of those factors. However, if the factor measured is inadequately measured, what does 
improvement imply? 

Further, if tlie constnicts puq^orted to be measured by this survey are valid for otJier 
institutions and tJie difference in constnicts reflects only ETSU students, to compare tlie mean for 
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these students to tliose of other institutions is not vaJid. h is suggested that further research should 
be conducted using students from ETSU to establish whether die results from this sample are a 
one-sample anomaly or indeed reflect accurately the constmcis for this population. Further 
research should also be conducted comparing samples from various institutions of higher education 
to detennine if the Six-Factor Tlieory Based Model adequately represents tliem. 



lb 
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Table 1 

Six-Factor Tlieory Based Model as Proposed by THEC 



Latent Variable Corresponding Questions 



Satisfaction 1 2 3 4 3 6 7 

Involvement 8 9 10 1112 13 14 15 16 

Personal Development 33 34 35 36 37 38 40 45 47 

Learning 39 41 42 43 44 46 48 49 50 5 1 52 

53 

Major Instruction/Advising 54 55 58 59 60 62 65 

Major Curriculum 56 57 6 1 63 64 66 67 68 69 



ERIC 



Table 2 

Six-Factor Inductive Model 
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Latent Variable Corresponding Questions 

Satisfaction 1 2 3 4 5 6 7 67 68 69 

Academic Involvement 89101112131415 

Analytical Problem Solving 44 46 48 5 1 53 

Cultural Awareness 34 39 49 50 52 2 

Personal Development 35 36 37 38 40 4 1 42 43 45 47 

Major 54 55 56 57 58 59 6 1 62 63 64 56 66 



Table 3 

Ten-Factor Inductive Model 
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Latent Variable 



Corresponding Questions 



Satisfaction 
Academic Involvement 
Comprehension 
Personal Development 
Major 

Cultural Awareness 
Analytical Problem Solving 
College Advising 
Degree Requirements 
Faculty Guidance 



1234567 

891011 12 13 14 15 16 

42 43 44 45 

35 36 37 38 40 41 47 

58 33 61 62 63 64 65 66 67 68 69 
34 39 49 50 52 

46 48 51 53 
54 55 
56 57 

59 60 
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Table 4 

Results of Comparative Models 



Model Non-redundant 

Elements of Z 


Factors 




d.f. 


CFia 


Six-Factor Theory Model 


1431 


6 


9181* 


1310 


0.75 


Six-Factor Liductive Model^ 


1431 


6 


8718* 


1310 


0.77 


Ten Factor Inductive ModeF 


1431 


10 


5739* 


1280 


0.86 


Refined Model 


990 


23d 


2604* 


862 


0.94 



Comparative Fit Index. The y} for the null model with 1431 elements of E is 33154 (1378 d.f.). 
For the refined model, the y} for the null model is 29553 (946 d.f.). 

model v^as developed using an exploratory factor analysis witli tlie tiie number of factors 
constrained to 6 (see Table 2). 

^lis model (see Table 3) v^as developed using an unconstrained exploratory factor analysis with 
standard extraction criteria (eigenvaluol). 

^llie Refined Model h;is 16 first order factors and 7 second order factors (see Figure 1 ). 
*Significant at probability of < .001 . 



ERIC 
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Table 5 

Total Effects of Y on ReFmed Model 



Satisfaction Research Learning Major 

Culture Personal Faculty 

Development 



Ql .330 

Q3 .462 

Q4 ,44 6 

Q5 .306 

Q7 .4 42 

Q6 .261 .176 
Q34 .433 
Q39 .52 9 

Q49 .401 
Q50 .404 
Q52 .444 

Q8 .462 

Q9 .683 
QIO .430 
Qll .681 
Q35 .374 
Q36 .441 
Q37 .416 
Q38 .441 
Q40 .416 
Q41 .455 
Q42 .368 
Q^3 .420 
Q44 .465 
Q^5 .351 
0-56 .441 
Q48 ,38 J 

Q51 .264 



2 1 
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Table 5 (cent.) 

Total Effects of Y on Refined Mode l 



Satisfaction Research Learning Major 

Culture Personal Faculty 

Development 



Q53 


. 395 




Q54 


. 631 




Q55 


. 609 




Q59 


. 664 




Q60 


. 681 




Q58 


. 260 


. 259 


Q56 




. 518 


Q57 




. 532 


Q61 




. 604 


Q62 




. 566 


Q63 




. 621 


Q64 




. 673 


Q65 




. 544 


Q68 




. 368 


Q69 




. 410 


Q67 




. 295 
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Figure 1: Hierarchical Model of the Enrolled Student Survey (Refined Model), 



CFA of a Student Satisfaction Survey 23 
References 

Banta, T,W. (1988). Assessment as an instrument of state funding policy. New 
Directions for Institutional Research. (59V 81-94. 

Banta, T.W., Rudolph, L., & Van Dyke, J. (1994, June). Performance funding comes of 
age in Tennessee. Paper presentea at the meeting of the American Association of Higher Education 
9tli Annual Conference on Assessment & Quality, Washington D.C. 

Boyer, C. (1988). General overview and background of educational accountability and 
assessment. In R. Jensen (Ed.) Developing Criteria for Institutional Effectiveness, (p. 8). Santa 
Ana CA: Rancho Santiago Community College. 

Burnley, C. (1993). 1992>1993 performance funding report. Johnson City: East 
Tennessee State University, Office of Outcomes Assessment. 

Davies, G. K. (1991). The influence of public policy on the quality of higher education. 
In D. H. Finfiter, R. G. Baldwin, and J. R. Thelm (Eds.), The uneasy public policy triangle in 
higher education (pp. 37-44). New York: MacMillan. 

DeYoung, A.J. (1988). Appalachian educational research in the national context. Where 
we have been and where we are gomg. Journal of the Appalachian Studies Association . 29 - 49. 

Ewell, P;T, (1994, June). Assessment as mvolvement: Wha t it is supposed to be all 
about. Paper presented at the meeting of the American Association of Higher Education 9th 
Annual Conference on Assessment & Quality, Washington D.C. 

Hawk, G. (1993). East Tennessee State University fact book . Johnson City: East 
Tennessee State University, Office of Institutional Research. 

James, L. (1994, June). Customer research: A tool to focus improvement. Paper 
presented at the meeting of the American Association of Higher Education 9th Annual Conference 
on Assessment & Quality, Washington D.C. 

Peters, R. (1994, June). Some snarks are boojums: Accountability and tlie end of higher 
education. Paper presented at the meeting of the American Association of Higher Education 9th 
Annud Conference on Assessment & Quality, Washington D.C. 

Pike, G.R, (1993, August). College students' levels of achievement. Paper presented at 
the meeting of the Tennessee Association for Institution < Research^ Nashville, TN. 



CFA of a Student Satisfaction Survey 24 

Spear, J.H. (1993, Spring). Education and the re-visioning of Appalachia. Now luid 
Then, 10 (1). 21-22. 

Young, O.R. (1993). Institutional effectiveness and its predictors: Tlie North Carolina 
cotrununitv college system. Unpublished doctoral dissertation, North Carolina State University, 
Raleigli. 



2o 



CFA of a Student Satisfaction Survey 25 

Footnotes 

^Tliese models weie assessed using standard confirmatory factor analytic procedures 

where ( 1 ) tlie variances of the ^s, the diagonal elements of the symmetric <[) matrix, are fixed at 1 .0 

for identification purposes, while the nonredundant off-diagonal elements of O (covariances 

among the factors) are free to correlate; (2) 0§ is a diagonal matrix (the errors are uncorrelated) 

with the diagonal elements estimated parameters, and (3) each observable indicator is free to load 
on only one ^ (each questionnaire item loads on a single factor.) For all three of these models, the 

variance/covariance matrix of the questionnaire iteins, Z, has 53(53+1 )/2=1431 nonredundant 
elements. 

2'rhese null models provide comparison infonnation needed to calculate the Comparative 

Fit Index in Table 4. This index is robust to sample size variations, and makes allowances for 
changing model complexities. Tlie null models used in tliis study are of the traditional fomi where 
tlie only estimated parameters are the variances of the ^s (diagonal elements of <l>), covariances 

ainong the questionnaire items are fixed at 0, and tlie error matrix, 0§ is constrained to be a null 

matrix. 



